feat: require pgmq 1.5.1+ and remove compatibility layer #301

jumski · 2025-11-02T22:04:30Z

Summary

This PR removes the pgmq 1.4.x compatibility layer and requires pgmq 1.5.1 or higher. It eliminates deprecated functions, removes backported SQL code, and leverages new PGMQ features including message headers support.

⚠️ BREAKING CHANGE: This version requires pgmq 1.5.0 or higher and will NOT work with pgmq 1.4.x.

Changes

PGMQ Version Requirement

Minimum version: pgmq 1.5.0+ (tested with 1.5.1)
Migration guard: Added compatibility check in migration that fails early with clear error message if pgmq < 1.5.0 is detected
Affected packages: @pgflow/core, @pgflow/edge-worker

Removed Compatibility Layer

Backported Functions Removed

pgflow.read_with_poll (68 lines) - Removed backport, now uses native pgmq.read_with_poll directly
pgflow.create_realtime_partition - Removed deprecated partitioning function
Test file removed: supabase/tests/realtime/create_realtime_partition.test.sql (83 lines)

Function Updates

pgflow.set_vt_batch - Updated to return headers column from pgmq 1.5.0+:
- Changed return type from SETOF to TABLE with explicit column definitions
- Now returns: msg_id, read_ct, enqueued_at, vt, message, headers
- Added comprehensive pgTAP test for headers handling

Migration

File: pkgs/core/supabase/migrations/20251102201302_pgflow_upgrade_pgmq_1_5_1.sql

The migration includes:

Compatibility check - Verifies pgmq.message_record has headers column
Clear error message - Guides users to upgrade pgmq if version is incompatible
Function replacement - Drops and recreates set_vt_batch with new signature

Code Updates

TypeScript

PgflowSqlClient.ts - Changed queue polling to use pgmq.read_with_poll instead of pgflow.read_with_poll
Queue.ts - Updated queue reading logic
Database types - Regenerated to reflect new function signatures and removed deprecated functions
Test fixtures - Added headers field to message record types

SQL Tests

Updated 11 test files to remove calls to deprecated functions:

Removed pgflow.create_realtime_partition() calls from all realtime tests
Simplified test setup by using native PGMQ functions
Added new test: supabase/tests/set_vt_batch/headers_handling.test.sql

Documentation

README.md - Enhanced documentation with:
- Clarified root vs dependent map step behavior
- Improved formatting and examples
- Added edge case documentation
- Better explanation of map step input handling
BREAKING CHANGE notice in changeset explaining:
- Version requirement
- Migration path for Supabase users
- Self-hosting upgrade requirements

Migration Management Skill

Updated .claude/skills/migration-management/SKILL.md with additional troubleshooting guidance

Migration Guide

For Supabase Users

Recent Supabase versions include pgmq 1.5.0+ by default. Simply upgrade pgflow:

# Upgrade will apply the migration automatically
pnpm nx migrate @pgflow/core

The migration will verify pgmq compatibility and fail with a clear message if your pgmq version is too old.

For Self-Hosted Users

Before upgrading pgflow:

Verify your pgmq version:
If extversion < 1.5.0, upgrade pgmq first:
Then upgrade pgflow:

Benefits

Simplified codebase: Removed 200+ lines of compatibility code
Native PGMQ features: Direct access to new pgmq 1.5.0+ capabilities
Message headers: Support for metadata propagation (foundation for future features)
Cleaner tests: Removed deprecated function calls across test suite
Better error messages: Migration fails early with actionable guidance

Verified Fixes

This PR verifies that upstream issues in Supabase Realtime have been resolved:

Janitor's 10-minute delay causes silent failures in local development supabase/realtime#1369 - The Janitor's 10-minute startup delay that caused silent failures with realtime.send() after database resets has been fixed in recent Supabase versions
With the fix in place, pgflow no longer needs the create_realtime_partition workaround function that manually created partitions before tests
Removal of pgflow.create_realtime_partition and its test file confirms the upstream fix is stable and working as expected

Testing

All pgTAP tests updated and passing
New test added for headers handling
Migration tested on both compatible (1.5.1) and incompatible (1.4.4) versions
Edge worker integration tests updated with headers support

Dependencies

Requires: PR chore: upgrade PostgreSQL 17 and Supabase CLI 2.54.11 #300 (PostgreSQL 17 and Supabase CLI 2.54.11 upgrade)
Breaking: Incompatible with pgmq < 1.5.0

Related Files

Changed:

37 files changed, 629 insertions(+), 926 deletions(-)
Net reduction of ~300 lines of code

Key files:

Migration: pkgs/core/supabase/migrations/20251102201302_pgflow_upgrade_pgmq_1_5_1.sql
Schema: pkgs/core/schemas/0110_function_set_vt_batch.sql
Client: pkgs/core/src/PgflowSqlClient.ts
Worker: pkgs/edge-worker/src/queue/Queue.ts

changeset-bot · 2025-11-02T22:04:35Z

🦋 Changeset detected

Latest commit: 20fcd48

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 6 packages

Name	Type
@pgflow/core	Minor
@pgflow/edge-worker	Minor
pgflow	Minor
@pgflow/client	Minor
@pgflow/example-flows	Minor
@pgflow/dsl	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

jumski · 2025-11-02T22:04:46Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

merge:queue - adds this PR to the back of the merge queue
hotfix:queue - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

nx-cloud · 2025-11-02T22:05:40Z

View your CI Pipeline Execution ↗ for commit 20fcd48

Command	Status	Duration	Result
`nx affected -t verify-exports --base=origin/mai...`	✅ Succeeded	3s	View ↗
`nx affected -t build --configuration=production...`	✅ Succeeded	30s	View ↗
`nx affected -t lint typecheck test --parallel -...`	✅ Succeeded	7m 1s	View ↗
`nx affected -t test:e2e --parallel --base=origi...`	✅ Succeeded	5m 10s	View ↗

☁️ Nx Cloud last updated this comment at 2025-11-17 18:22:20 UTC

pkgs/core/supabase/migrations/20251102201302_pgflow_upgrade_pgmq_1_5_1.sql

graphite-app · 2025-11-03T11:13:48Z

pkgs/core/src/database-types.ts

+        Args: { msg_ids: number[]; queue_name: string; vt_offsets: number[] }
+        Returns: {
+          enqueued_at: string
+          headers: Json
+          message: Json
+          msg_id: number
+          read_ct: number
+          vt: string
+        }[]


The headers field in the return type is typed as Json but should be Json | null since the JSONB column in PostgreSQL can be NULL. This type mismatch could cause runtime errors when TypeScript code receives null headers from messages that were sent without headers.

set_vt_batch: { Args: { msg_ids: number[]; queue_name: string; vt_offsets: number[] } Returns: { enqueued_at: string headers: Json | null // Should allow null message: Json msg_id: number read_ct: number vt: string }[] }

This aligns with how headers is typed elsewhere in the codebase (line 784) and matches the actual database behavior where headers can be NULL for messages sent without headers.

Suggested change

Args: { msg_ids: number[]; queue_name: string; vt_offsets: number[] }

Returns: {

enqueued_at: string

headers: Json

message: Json

msg_id: number

read_ct: number

vt: string

}[]

Args: { msg_ids: number[]; queue_name: string; vt_offsets: number[] }

Returns: {

enqueued_at: string

headers: Json | null

message: Json

msg_id: number

read_ct: number

vt: string

}[]

Spotted by Graphite Agent

Is this helpful? React 👍 or 👎 to let us know.

pkgs/core/supabase/migrations/20251104080523_pgflow_upgrade_pgmq_1_5_1.sql

…test references - Deleted the create_realtime_partition SQL function and related comments - Updated test scripts to no longer call create_realtime_partition - Replaced pgflow.read_with_poll with pgmq.read_with_poll in client code - Ensured tests focus on existing functionality without partition creation - Minor adjustments to test setup for consistency and clarity

github-actions · 2025-11-17T18:23:08Z

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-301.pgflow.pages.dev

📝 Details:

Branch: chore-remove-backports-and-update-to-new-pgmq
Commit: 5687cf304b8f4fc42e90f87080313ce3520670ed
View Logs

_Last updated: _

jumski mentioned this pull request Nov 2, 2025

chore: upgrade PostgreSQL 17 and Supabase CLI 2.54.11 #300

Open

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from b531d64 to c9b5c0e Compare November 2, 2025 23:01

graphite-app bot reviewed Nov 2, 2025

View reviewed changes

pkgs/core/supabase/migrations/20251102201302_pgflow_upgrade_pgmq_1_5_1.sql Show resolved Hide resolved

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch 2 times, most recently from 4a21116 to 361b043 Compare November 3, 2025 10:09

jumski force-pushed the chore-update-supabase-versions branch from 9f66fe6 to dfb9cbe Compare November 3, 2025 11:10

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 361b043 to 6bf098a Compare November 3, 2025 11:10

graphite-app bot reviewed Nov 3, 2025

View reviewed changes

jumski force-pushed the chore-update-supabase-versions branch from dfb9cbe to e5e3dfa Compare November 4, 2025 08:14

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 6bf098a to 53aaf5c Compare November 4, 2025 08:14

jumski force-pushed the chore-update-supabase-versions branch from e5e3dfa to 8f99711 Compare November 6, 2025 09:24

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch 2 times, most recently from 968c2ae to 05b808d Compare November 6, 2025 12:04

graphite-app bot reviewed Nov 6, 2025

View reviewed changes

pkgs/core/supabase/migrations/20251104080523_pgflow_upgrade_pgmq_1_5_1.sql Outdated Show resolved Hide resolved

jumski temporarily deployed to preview November 6, 2025 17:02 — with GitHub Actions Inactive

jumski force-pushed the chore-update-supabase-versions branch from 8f99711 to b05ba56 Compare November 9, 2025 16:06

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 05b808d to b05b2ae Compare November 9, 2025 16:06

jumski force-pushed the chore-update-supabase-versions branch from b05ba56 to 06a26d1 Compare November 10, 2025 22:17

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch 2 times, most recently from e6a446e to 1d22df3 Compare November 10, 2025 23:27

jumski force-pushed the chore-update-supabase-versions branch from 06a26d1 to e1462be Compare November 10, 2025 23:27

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 1d22df3 to ff95e03 Compare November 10, 2025 23:44

jumski force-pushed the chore-update-supabase-versions branch 2 times, most recently from 3eb2f52 to 1deb956 Compare November 11, 2025 11:38

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from ff95e03 to 00b1670 Compare November 11, 2025 11:38

jumski force-pushed the chore-update-supabase-versions branch from 1deb956 to eba7284 Compare November 11, 2025 14:39

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 00b1670 to 508c6ac Compare November 11, 2025 14:39

jumski force-pushed the chore-update-supabase-versions branch from eba7284 to f8447bd Compare November 12, 2025 22:58

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 508c6ac to b1b578b Compare November 12, 2025 22:58

jumski force-pushed the chore-update-supabase-versions branch from f8447bd to c35fab0 Compare November 12, 2025 23:10

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from b1b578b to 3af7bb9 Compare November 12, 2025 23:10

jumski force-pushed the chore-update-supabase-versions branch from c35fab0 to fb7da9f Compare November 16, 2025 21:47

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 3af7bb9 to eff7892 Compare November 16, 2025 21:47

jumski force-pushed the chore-update-supabase-versions branch from fb7da9f to b5e440f Compare November 17, 2025 08:02

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from eff7892 to 2c72979 Compare November 17, 2025 08:02

jumski force-pushed the chore-update-supabase-versions branch from b5e440f to 6b652ef Compare November 17, 2025 09:41

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 2c72979 to c0e2974 Compare November 17, 2025 09:41

jumski force-pushed the chore-update-supabase-versions branch from 6b652ef to e6b4dbc Compare November 17, 2025 11:11

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from c0e2974 to efa0ec2 Compare November 17, 2025 11:11

jumski temporarily deployed to preview November 17, 2025 11:20 — with GitHub Actions Inactive

jumski temporarily deployed to preview November 17, 2025 13:23 — with GitHub Actions Inactive

jumski temporarily deployed to preview November 17, 2025 13:47 — with GitHub Actions Inactive

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from efa0ec2 to 5e2dfb1 Compare November 17, 2025 15:32

jumski force-pushed the chore-update-supabase-versions branch from e6b4dbc to 229a71e Compare November 17, 2025 15:32

jumski temporarily deployed to preview November 17, 2025 15:41 — with GitHub Actions Inactive

jumski force-pushed the chore-remove-backports-and-update-to-new-pgmq branch from 5e2dfb1 to 20fcd48 Compare November 17, 2025 17:39

jumski force-pushed the chore-update-supabase-versions branch from 229a71e to 211fd84 Compare November 17, 2025 17:39

jumski temporarily deployed to preview November 17, 2025 18:21 — with GitHub Actions Inactive

This was referenced Nov 17, 2025

docs: add a 0.8.0 release post and update README.md #390

Open

chore: update deno to 2.1.4 #391

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: require pgmq 1.5.1+ and remove compatibility layer #301

feat: require pgmq 1.5.1+ and remove compatibility layer #301

jumski commented Nov 2, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Nov 2, 2025 •

edited

Loading

Uh oh!

jumski commented Nov 2, 2025 •

edited

Loading

Uh oh!

nx-cloud bot commented Nov 2, 2025 •

edited

Loading

Uh oh!

Uh oh!

graphite-app bot Nov 3, 2025

Uh oh!

Uh oh!

github-actions bot commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: require pgmq 1.5.1+ and remove compatibility layer #301

Are you sure you want to change the base?

feat: require pgmq 1.5.1+ and remove compatibility layer #301

Conversation

jumski commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

PGMQ Version Requirement

Removed Compatibility Layer

Backported Functions Removed

Function Updates

Migration

Code Updates

TypeScript

SQL Tests

Documentation

Migration Management Skill

Migration Guide

For Supabase Users

For Self-Hosted Users

Benefits

Verified Fixes

Testing

Dependencies

Related Files

Uh oh!

changeset-bot bot commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

jumski commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

nx-cloud bot commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

graphite-app bot Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Nov 17, 2025

🔍 Preview Deployment: Website

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jumski commented Nov 2, 2025 •

edited

Loading

changeset-bot bot commented Nov 2, 2025 •

edited

Loading

jumski commented Nov 2, 2025 •

edited

Loading

nx-cloud bot commented Nov 2, 2025 •

edited

Loading